{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01135e5b",
   "metadata": {},
   "source": [
    "Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: Apache-2.0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e147f141",
   "metadata": {},
   "source": [
    "# Modeling molecular SMILES data with Amazon Neptune and RDKit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b87be86",
   "metadata": {},
   "source": [
    "This notebook walks through the process of modeling chemical structures using Amazon Neptune. It includes transforming a chemical compound represented as a SMILES string into graph data, with nodes representing individual atoms and edges representing the bonds between atoms.This notebook then closes with a visualization and exploration of the molecule caffeine."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9716ef94",
   "metadata": {},
   "source": [
    "   - [Background](#Background)\n",
    "\n",
    "   - [Solution Overview](#Solution-Overview)\n",
    "\n",
    "   - [Package Setup](#Package-Setup)\n",
    "\n",
    "   - [Graph Data Model](#Graph-Data-Model)\n",
    "\n",
    "   - [RDKit Processing](#RDKit-Processing)\n",
    "\n",
    "   - [Amazon Neptune Data Upload](#Data-Upload)\n",
    "\n",
    "   - [Basic Visualization & Queries](#Basic-Visualization-&-Queries)\n",
    "\n",
    "   - [Clean Up](#Clean-Up)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23409061",
   "metadata": {},
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63016f6c",
   "metadata": {},
   "source": [
    "   Modeling chemical structures can be a complex and tedious process, even with the help of modern programs and technology. The ability to explore chemical structures at the most fundamental level of atoms and the bonds that connect them is an essential process in [drug discovery, pharmaceutical research](https://pubs.acs.org/doi/abs/10.1021/acs.jcim.0c00947), and [chemical engineering](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8130509/). By infusing chemical research with technology, researchers can expedite outcome timelines, identify hidden relationships, and overall simplify a traditionally complex process. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9bfdfa6",
   "metadata": {},
   "source": [
    "## Solution Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33e4d4ac",
   "metadata": {},
   "source": [
    "In order to integrate technology into the analysis of chemical structures, molecules themselves must first be represented in a machine-readable format, such as [SMILES (simplified molecular-input line-entry system)](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system). **SMILES** format strings are the industry standard in representing molecular structures. The SMILES format enables the relationships between atoms in a molecular structure to be conveyed as a machine processable string. The SMILES format is not all encompassing, leaving out details such as certain polarities and bond properties. However, SMILES does enable powerful analysis at scale of different structures. \n",
    "\n",
    "Using Amazon Neptune and the open-source chemical-Informaics software package [RDKit](https://www.rdkit.org/), SMILES format data can be ingested, processed, and converted into nodes and edges in a property graph. Modeling molecular structures in a graph database allows for powerful custom visualization and manipulation at the scale demanded by pharmaceutical applications. Utilizing a graph database such as Neptune allows users to compare millions of molecules with millions of associated interactions. Additionally, the fully managed and serverless infrastructure allows experts with backgrounds in biology and chemistry to focus primarily on the research outcomes of their graph data, avoiding the undifferentiated heavy lifting of managing a complex graph database infrastructure.\n",
    "\n",
    "This walkthrough follows the process of converting a singular SMILES string, **caffeine**, to graph data in Neptune. However, the process will work for any SMILES format string you would like to use. We’re sourcing the string for caffeine from the [National Library of Medicine](https://pubchem.ncbi.nlm.nih.gov/compound/2519#section=InChI), which maintains a public dataset of many chemical structures [**CN1C=NC2=C1C(=O)N(C(=O)N2C)C**].\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79e81e96",
   "metadata": {},
   "source": [
    "We also use the open-source cheminformatics package [RDKit](https://github.com/rdkit/rdkit), Python-based data science package [Pandas](https://pandas.pydata.org/), and [AWS SDK for pandas (awswrangler)](https://aws-sdk-pandas.readthedocs.io/en/stable/). RDKit has a strong community and a great number of chem-informatics utilities; we’ll only be exploring a small portion for this post. Pandas is an open-source Python-based data science toolkit with large community support. The AWS SDK for pandas provides a large set of tools to help AWS services interact with the pandas library."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49cf299",
   "metadata": {},
   "source": [
    "## Package Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e1c8a9d",
   "metadata": {},
   "source": [
    "The first step in modeling a chemical structure as graph data is importing the required packages. Here we will be using **RDKit**, **Pandas**, and **awswrangler**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb5f3b98",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install rdkit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "440c97e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install awswrangler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a5625f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from rdkit import Chem\n",
    "import pandas as pd\n",
    "import awswrangler as wr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24e2d68d",
   "metadata": {},
   "source": [
    "## Graph Data Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "536a3491",
   "metadata": {},
   "source": [
    "There are a few different options for graph query languages and their associated data models when working with Neptune; in this case we’re using **Apache TinkerPop’s Gremlin**. We are opting for Gremlin here due to its intuitive nature and easy to learn syntax. The cell below is defining a dictionary object for both the nodes and edges of our graph. Within each dictionary object is a set of properties we will gather from our caffeine molecule in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a8cd157",
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_dict = {'~id':[],\n",
    "                '~label':[],\n",
    "                'idx':[],\n",
    "                'atomicNumber':[],\n",
    "                'isAromatic': []\n",
    "               }\n",
    "               \n",
    "edges_dict = {'~id':[],\n",
    "                '~label':[],\n",
    "                '~from':[],\n",
    "                '~to':[],\n",
    "               }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d049b9e6",
   "metadata": {},
   "source": [
    "## RDKit Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21846f4c",
   "metadata": {},
   "source": [
    "This section is where the chemical computing magic happens. We use the `rdkit` package installed earlier in the graph notebook to decompose our chemical structure into lists of **nodes (atoms)** and **edges (bonds)**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69d0e0d",
   "metadata": {},
   "source": [
    "First, we want to declare our SMILES string for the caffeine molecule as a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33a9fdf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "caffeine_smiles = 'CN1C=NC2=C1C(=O)N(C(=O)N2C)C'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cacd427b",
   "metadata": {},
   "source": [
    "Next, to obtain a molecule-type object from the RDKit package, we need to use the below call to the `Chem` library from RDKit imported earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "208de8b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "mol = Chem.MolFromSmiles(caffeine_smiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45119037",
   "metadata": {},
   "source": [
    "Let’s see a visual of our work so far, run the following cell to output a 2D picture of our molecule below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7260f5c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "mol"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1932568",
   "metadata": {},
   "source": [
    "To recap what we just did, first we declared our SMILES string for caffeine as the variable `caffeine_smiles`. Next, we used the `Chem.MolFromSmiles` function from RDKit to turn the SMILES into a `mol` type object defined by RDKit. Finally, we returned the `mol` type object which resulted in a 2D image of the molecular structure for caffeine that we were working with."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cbd5ec2",
   "metadata": {},
   "source": [
    "Now we need to iterate through each atom and bond within the `mol` object outputted from RDKit. While iterating through each atom and bond, we use the graph data model we declared earlier, storing properties of each inside the data model. Feel free to dive deeper into the `mol.GetAtoms()` and `mol.GetBonds()` function calls on your own - we are only exploring a small subset of their functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3b030e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "for atom in mol.GetAtoms():\n",
    "    nodes_dict['~id'].append('Node-'+ caffeine_smiles + str(atom.GetIdx()))\n",
    "    nodes_dict['~label'].append(atom.GetSymbol())\n",
    "    nodes_dict['idx'].append(atom.GetIdx())\n",
    "    nodes_dict['atomicNumber'].append(atom.GetAtomicNum())\n",
    "    nodes_dict['isAromatic'].append(atom.GetIsAromatic())\n",
    "\n",
    "for bond in mol.GetBonds():\n",
    "    edges_dict['~id'].append('edge-'+ caffeine_smiles + str(bond.GetBeginAtomIdx()) + str(bond.GetEndAtomIdx()))\n",
    "    edges_dict['~label'].append(str(bond.GetBondType()))\n",
    "    edges_dict['~from'].append('Node-' + caffeine_smiles + str(bond.GetBeginAtomIdx()))\n",
    "    edges_dict['~to'].append('Node-' + caffeine_smiles + str(bond.GetEndAtomIdx()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09038090",
   "metadata": {},
   "source": [
    "Several different RDKit functions are in this portion of code, so let’s break it down piece by piece: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caeeaeb3",
   "metadata": {},
   "source": [
    "•\tFor the `~id` field of both nodes and edges, we combine the data type `Node` or `Edge`, the SMILES string, and the unique index for the atom"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e8287cf",
   "metadata": {},
   "source": [
    "•\tFor the `~label` field, we use the chemical symbol for nodes, and the bond type for the edges"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19bf0671",
   "metadata": {},
   "source": [
    "•\tThe fields `~from` and `~to` for the edges (bonds) are constructed by combining the prefix `Node-` with the SMILES string, and the respective beginning and ending atoms that the bond connects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0da3d8f8",
   "metadata": {},
   "source": [
    "•\tThe additional fields for the nodes (atoms) in the graph model are the atom’s unique ID within the molecule, its atomic number, and if it is aromatic or not"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ccc7a32",
   "metadata": {},
   "source": [
    "Note that you can extract several atomic properties for a given SMILES string in RDKit and add them as additional fields for a given atom or bond. We don’t list them all in this post, but you can explore additional fields for both the atoms and bonds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcbf50b9",
   "metadata": {},
   "source": [
    "Finally, we want to use **Pandas** to transform our data-poulated dictionaries into pandas data frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab3a8fa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_df = pd.DataFrame.from_dict(nodes_dict)\n",
    "edges_df = pd.DataFrame.from_dict(edges_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35572287",
   "metadata": {},
   "source": [
    "let's check the results of our work so far. Running the cells below should return data frames for both the edges and nodes of our caffeine molecule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f8df5de",
   "metadata": {},
   "outputs": [],
   "source": [
    "edges_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1096a312",
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da724549",
   "metadata": {},
   "source": [
    "## Amazon Neptune Data Upload"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ead358b",
   "metadata": {},
   "source": [
    "Now that we have successfully decomposed our caffeine SMILES string into individual atoms and bonds, the next step is to load our data into the Neptune database itself. This will be much simpler than loading data from an external source because our data is already inside the graph notebook environment. In order to write our data to the Neptune database, we will be using the **AWS SDK for pandas**, also known as **awswrangler**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "271ac986",
   "metadata": {},
   "source": [
    "First, we need to check our notebook configuration to gather the host endpoint for our cluster. Running the cell below will provide that information, along with other important details about our Neptune database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c779af",
   "metadata": {},
   "outputs": [],
   "source": [
    "%graph_notebook_config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66745fed",
   "metadata": {},
   "source": [
    "Find the `host` field from the above output and copy & paste the string into the cell below where it says `'[INSERT YOUR HOST HERE]'`. Also, check the port number above, ensure that the port number above is the same as the second parameter in the cell below. The default port number should be `8182`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7343a1e7",
   "metadata": {},
   "source": [
    "Run cell below once the host & port is properly copied from your graph configuration output above. This cell is simply using a command from the Neptune section of the awswrangler library to establish a connection to your Neptune instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b5da429",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = wr.neptune.connect(\"INSERT YOUR HOST HERE\", 8182, iam_enabled=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d26e71fd",
   "metadata": {},
   "source": [
    "The next two cells use the `.to_property_graph` functions within awswrangler to insert both the node & edge data frames we created earlier into our Neptune database. Both cells should return a `\"True\"` upon success."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60f6dcf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "wr.neptune.to_property_graph(client, df=nodes_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e06f3138",
   "metadata": {},
   "outputs": [],
   "source": [
    "wr.neptune.to_property_graph(client, df=edges_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8064ef96",
   "metadata": {},
   "source": [
    "After receiving a `\"True\"` output from both cells, you are finished with processing your SMILES molecule string. Now you can move onto visualizing your molecule as graph data. If you wish to add additional compounds to your graph database, you can return to the start of the [RDKit Processing](#RDKit-Processing) section and simply redo the process with a different SMILES string."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da70417",
   "metadata": {},
   "source": [
    "## Basic Visualization & Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb19689b",
   "metadata": {},
   "source": [
    "Now that we have processed and loaded our molecular data, let's visualize the results of our efforts so far. The below cell uses a Gremlin query to traverse outwards from nodes which are labeled as **'C'** (*for the element Carbon*), giving us a picture of the overall structure of our molecule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dbb8414",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%gremlin -p v,ine,outv,oute,inv,oute,inv,oute,inv,oute,inv\n",
    "g.V().has('~label','C').repeat(outE().inV()).emit().times(5).path().by(valueMap(true))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "553a58d8",
   "metadata": {},
   "source": [
    "You should receive an interactive view of the 2D image RDKit produced earlier as the output of the above cell. Feel free to explore the structure and compare it to other visualizations you might be able to find of your molecule. Also, now that our molecule is persisted as graph data, you can manipulate, edit, and add additional data to your molecular structure as you see fit."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bc4961d",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7dd3d35",
   "metadata": {},
   "source": [
    "In this guide, you ingested and parsed a **SMILES** format molecular data string with **RDKit** and\n",
    "uploaded the individual atoms and bonds as graph data to **Amazon Neptune**. You can replicate this\n",
    "process at *scale* to accommodate large datasets containing many SMILES strings. You can test\n",
    "this yourself by following the steps for any SMILES string of your choice. With the molecular\n",
    "data broken into individual atoms and bonds in Neptune, you can connect this data to **custom\n",
    "bioinformatics applications**, **chemical computing systems**, and **research software environments**.\n",
    "You can take this solution even further by integrating [Amazon Neptune ML](https://aws.amazon.com/blogs/database/how-to-get-started-with-neptune-ml/) to gain the ability to\n",
    "predict the connections and properties of your molecules."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b360fb55",
   "metadata": {},
   "source": [
    "**See these other resources below to learn more about Amazon Neptune's role in Healthcare & Life Sciences:**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fefa124d",
   "metadata": {},
   "source": [
    "- [Accelerating drug discovery through knowledge graphs](https://aws.amazon.com/blogs/industries/accelerating-drug-discovery-through-knowledge-graph/)\n",
    "\n",
    "- [Analyze healthcare FHIR data with Amazon Neptune](https://aws.amazon.com/blogs/database/analyze-healthcare-fhir-data-with-amazon-neptune/)\n",
    "\n",
    "- [Building Amazon Neptune based MedDRA terminology mapping for pharmacovigilance and adverse event reporting](https://aws.amazon.com/blogs/industries/building-amazon-neptune-based-meddra-terminology-mapping-for-pharmacovigilance-and-adverse-event-reporting/)\n",
    "\n",
    "- [Building and querying the AWS COVID-19 knowledge graph](https://aws.amazon.com/blogs/database/building-and-querying-the-aws-covid-19-knowledge-graph/)\n",
    "\n",
    "- [General Neptune Developer Resources](https://aws.amazon.com/neptune/developer-resources/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f7c99a3",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b35cd1c",
   "metadata": {},
   "source": [
    "*WILL DELETE MOLECULE DATA*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d267a23",
   "metadata": {},
   "source": [
    "Run the cell below to delete the data in the graph if you want to clean your graph database storage. This iterates through the `nodes_df`, so be sure to adjust accordingly if you have added any of your own additional molecules or edits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "363ea618",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in nodes_df['~id']:\n",
    "    wr.neptune.execute_gremlin(client, \"g.V().has('~id', '\"+i+\"').drop();\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
